Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TT-922, TT-943] Migrate automation and log poller tests to Seth #12688

Merged
merged 122 commits into from
May 13, 2024

Conversation

Tofel
Copy link
Contributor

@Tofel Tofel commented Apr 4, 2024

Migrated:

  • automation load test
  • automation chaos test
  • keeper benchmark test
  • automation smoke tests
  • keeper smoke tests
  • log poller smoke tests

Excluded from ever being migrated:

  • automation reorg test (we are changing our approach to this type of tests)

Important details:

  • no existing actions or contract wrappers were deleted or modified with exception of NewAutomationTestDocker(), which is used only by automation load test; instead Seth-based actions were created and used in migrated tests
  • all contract wrappers using EVMClient were renamed to Legacy*
  • all keeper/automation/log poller tests now require at least 1 ephemeral key, ideally more of them, since contract deployments and setup is parallelised to speed it up
  • it's disallowed to run tests on live networks with ephemeral keys (if you configure your tests to use them anyway code will disable them)
  • I added an universal concurrent executor to use common logic for parallel deployments

@Tofel Tofel changed the title [TT-922] Migrate automation some tests to Seth [TT-922] Migrate automation tests to Seth Apr 11, 2024
kalverra
kalverra previously approved these changes May 10, 2024
@Tofel Tofel enabled auto-merge May 10, 2024 14:40
anirudhwarrier
anirudhwarrier previously approved these changes May 10, 2024
@Tofel Tofel added this pull request to the merge queue May 10, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 10, 2024
@Tofel Tofel added this pull request to the merge queue May 10, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 10, 2024
@Tofel Tofel dismissed stale reviews from anirudhwarrier and kalverra via 6591af7 May 10, 2024 18:03
@Tofel Tofel enabled auto-merge May 10, 2024 18:04
@cl-sonarqube-production
Copy link

@Tofel Tofel added this pull request to the merge queue May 10, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 10, 2024
@Tofel Tofel added this pull request to the merge queue May 13, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 13, 2024
@Tofel Tofel added this pull request to the merge queue May 13, 2024
Merged via the queue into develop with commit f72c899 May 13, 2024
107 checks passed
@Tofel Tofel deleted the tt_922_automation_seth branch May 13, 2024 07:36
Tofel added a commit that referenced this pull request May 13, 2024
)

* WIP#1

* fix load test

* lints

* lints again

* uncomment tests

* fix logtrigger mercury test

* set privilege before starting consumer

* WIP#2

* WIP#2 - with high concurrency upkeep registration fails

* more or less working

* WIP#4

* small fixes

* fix lints

* use latest Seth

* Migrated keeper benchmark, kind of

* migrate keeper smoke tests

* bunch of optimisation

* migrate automation chaos test

* migrate log poller tests to Seth

* use anysynced key with log poller

* fix some of failing smoke tests

* fix go mod

* use correct Seth version

* better automation some test structure

* better automation some test structure

* go mod

* fix issues with missing LINK funds

* fix keeper test

* fix automation test

* clean up, comments, migrate keeper benchmark test

* mend

* use latest Seth

* cleanups, comments, etc

* clean ups, automation tests + keeper tests now always test multiple contracts

* use newer action version

* debug results when upkeeps fails

* run automation tests with post-failure debug

* run automation tests with post-failure debug

* better staleness report

* more debug for basic automation tests

* more assertions

* try with sleep

* increase performance check timeout to 3m

* separate config from consumer start

* more debug for performance chcek

* try with just 1 contract

* one more clean up

* run keeper tests in CI to test them

* solve automation load test registration failures

* subscribe to headers in benchmark test to observe upkeeps

* centralised ephemeral keys on testnet check, fixed keeper smoke tests, added root key buffer check

* do not fail, when on testnet and ephemeral keys are !=0, set them 0 instead

* don't fail in test config, display warning if eph addresses are set and set them to 0

* fix possible nil pointer, remove block range timeout, mutex for error appending

* update seth version

* use buffered channel to read subscription headers from EVM node

* benchmark test increase RR mem and set DEBUG log level

* use CTF's concurrent executor to deploy contracts in parallel

* change architecture so that we have only 1 subscription and then we fan out the header to all goroutines; use SafeEVMHeader

* benchmark test increase geth disk and gaslimit

* decrease log level, latest seth and ctf

* update CTF dependency

* use concurrent executor in automation load test

* remove local replace

* add header subscription reconnect handling

* print return funds error

* limit automation load test deployment concurrency

* latest Seth + fix gun

* do not run keeper smoke tests on each commit anymore

* add one more error message

* use tagged CTF

* use latest Seth

* use latest Seth

* fix go.mod

* forget about smart reconnect, just do it in a loop

* use latest Seth

* check if context has error before executing any of contract calling wrapper functions

* prettify code + latest Seth

* update Seth

* optional retrying of read-only contract calls; used only by keeper benchmark test

* update Seth, estimate gas for funds sending

* fix new automation test, update Seth to v1.0.7

* increase timeout

* fix max price smoke test

* fix CreateOCRKeeperJobs for v2.0

* remove resolved TODO

* code review changes

* code review changes

* code review changes vol 2

* use CTF release version

* bump mod version

---------

Co-authored-by: anirudhwarrier <12178754+anirudhwarrier@users.noreply.github.com>
Tofel added a commit that referenced this pull request May 13, 2024
)

* WIP#1

* fix load test

* lints

* lints again

* uncomment tests

* fix logtrigger mercury test

* set privilege before starting consumer

* WIP#2

* WIP#2 - with high concurrency upkeep registration fails

* more or less working

* WIP#4

* small fixes

* fix lints

* use latest Seth

* Migrated keeper benchmark, kind of

* migrate keeper smoke tests

* bunch of optimisation

* migrate automation chaos test

* migrate log poller tests to Seth

* use anysynced key with log poller

* fix some of failing smoke tests

* fix go mod

* use correct Seth version

* better automation some test structure

* better automation some test structure

* go mod

* fix issues with missing LINK funds

* fix keeper test

* fix automation test

* clean up, comments, migrate keeper benchmark test

* mend

* use latest Seth

* cleanups, comments, etc

* clean ups, automation tests + keeper tests now always test multiple contracts

* use newer action version

* debug results when upkeeps fails

* run automation tests with post-failure debug

* run automation tests with post-failure debug

* better staleness report

* more debug for basic automation tests

* more assertions

* try with sleep

* increase performance check timeout to 3m

* separate config from consumer start

* more debug for performance chcek

* try with just 1 contract

* one more clean up

* run keeper tests in CI to test them

* solve automation load test registration failures

* subscribe to headers in benchmark test to observe upkeeps

* centralised ephemeral keys on testnet check, fixed keeper smoke tests, added root key buffer check

* do not fail, when on testnet and ephemeral keys are !=0, set them 0 instead

* don't fail in test config, display warning if eph addresses are set and set them to 0

* fix possible nil pointer, remove block range timeout, mutex for error appending

* update seth version

* use buffered channel to read subscription headers from EVM node

* benchmark test increase RR mem and set DEBUG log level

* use CTF's concurrent executor to deploy contracts in parallel

* change architecture so that we have only 1 subscription and then we fan out the header to all goroutines; use SafeEVMHeader

* benchmark test increase geth disk and gaslimit

* decrease log level, latest seth and ctf

* update CTF dependency

* use concurrent executor in automation load test

* remove local replace

* add header subscription reconnect handling

* print return funds error

* limit automation load test deployment concurrency

* latest Seth + fix gun

* do not run keeper smoke tests on each commit anymore

* add one more error message

* use tagged CTF

* use latest Seth

* use latest Seth

* fix go.mod

* forget about smart reconnect, just do it in a loop

* use latest Seth

* check if context has error before executing any of contract calling wrapper functions

* prettify code + latest Seth

* update Seth

* optional retrying of read-only contract calls; used only by keeper benchmark test

* update Seth, estimate gas for funds sending

* fix new automation test, update Seth to v1.0.7

* increase timeout

* fix max price smoke test

* fix CreateOCRKeeperJobs for v2.0

* remove resolved TODO

* code review changes

* code review changes

* code review changes vol 2

* use CTF release version

* bump mod version

---------

Co-authored-by: anirudhwarrier <12178754+anirudhwarrier@users.noreply.github.com>
github-merge-queue bot pushed a commit that referenced this pull request May 15, 2024
* add back functionality that checks cl node logs after tests for errors

* add comment, newer CTF version

* use CTF release version

* ignore expected errors in cl node logs

* fix go.mod

* fix load tests

* fix lints

* fix lints again

* fix another vrfv2 log error

* use CTF release version

* [TT-922, TT-943] Migrate automation and log poller tests to Seth (#12688)

* WIP#1

* fix load test

* lints

* lints again

* uncomment tests

* fix logtrigger mercury test

* set privilege before starting consumer

* WIP#2

* WIP#2 - with high concurrency upkeep registration fails

* more or less working

* WIP#4

* small fixes

* fix lints

* use latest Seth

* Migrated keeper benchmark, kind of

* migrate keeper smoke tests

* bunch of optimisation

* migrate automation chaos test

* migrate log poller tests to Seth

* use anysynced key with log poller

* fix some of failing smoke tests

* fix go mod

* use correct Seth version

* better automation some test structure

* better automation some test structure

* go mod

* fix issues with missing LINK funds

* fix keeper test

* fix automation test

* clean up, comments, migrate keeper benchmark test

* mend

* use latest Seth

* cleanups, comments, etc

* clean ups, automation tests + keeper tests now always test multiple contracts

* use newer action version

* debug results when upkeeps fails

* run automation tests with post-failure debug

* run automation tests with post-failure debug

* better staleness report

* more debug for basic automation tests

* more assertions

* try with sleep

* increase performance check timeout to 3m

* separate config from consumer start

* more debug for performance chcek

* try with just 1 contract

* one more clean up

* run keeper tests in CI to test them

* solve automation load test registration failures

* subscribe to headers in benchmark test to observe upkeeps

* centralised ephemeral keys on testnet check, fixed keeper smoke tests, added root key buffer check

* do not fail, when on testnet and ephemeral keys are !=0, set them 0 instead

* don't fail in test config, display warning if eph addresses are set and set them to 0

* fix possible nil pointer, remove block range timeout, mutex for error appending

* update seth version

* use buffered channel to read subscription headers from EVM node

* benchmark test increase RR mem and set DEBUG log level

* use CTF's concurrent executor to deploy contracts in parallel

* change architecture so that we have only 1 subscription and then we fan out the header to all goroutines; use SafeEVMHeader

* benchmark test increase geth disk and gaslimit

* decrease log level, latest seth and ctf

* update CTF dependency

* use concurrent executor in automation load test

* remove local replace

* add header subscription reconnect handling

* print return funds error

* limit automation load test deployment concurrency

* latest Seth + fix gun

* do not run keeper smoke tests on each commit anymore

* add one more error message

* use tagged CTF

* use latest Seth

* use latest Seth

* fix go.mod

* forget about smart reconnect, just do it in a loop

* use latest Seth

* check if context has error before executing any of contract calling wrapper functions

* prettify code + latest Seth

* update Seth

* optional retrying of read-only contract calls; used only by keeper benchmark test

* update Seth, estimate gas for funds sending

* fix new automation test, update Seth to v1.0.7

* increase timeout

* fix max price smoke test

* fix CreateOCRKeeperJobs for v2.0

* remove resolved TODO

* code review changes

* code review changes

* code review changes vol 2

* use CTF release version

* bump mod version

---------

Co-authored-by: anirudhwarrier <12178754+anirudhwarrier@users.noreply.github.com>

* add exception to other chaos log poller smoke tests

---------

Co-authored-by: anirudhwarrier <12178754+anirudhwarrier@users.noreply.github.com>
github-merge-queue bot pushed a commit that referenced this pull request May 20, 2024
* use latest Seth

* move test config to CTF

* use latest CTF that fixes eth2 genesis generation

* use latest Seth

* use tagged CTF version

* WIP#1

* WIP#2

* add automation-specifc node configuration to TOML

* add node TOML config for OCR forwarders smoke tests

* add CL node TOML configuration for keeper tests

* add CL node TOML config for log poller

* add OCR1/2 TOML config

* add VRF* CL node TOML config

* remove functional options for setting CL node config in tests, use TOML instead

* go mod tidy

* fix go mod

* fix test env builder

* increase node funding in runlog test

* increase vrf node funding

* add on-demand keeper smoke tests workflow

* run keeper smoke tests once

* fix int tests yaml?

* [TT-922, TT-943] Migrate automation and log poller tests to Seth (#12688)

* WIP#1

* fix load test

* lints

* lints again

* uncomment tests

* fix logtrigger mercury test

* set privilege before starting consumer

* WIP#2

* WIP#2 - with high concurrency upkeep registration fails

* more or less working

* WIP#4

* small fixes

* fix lints

* use latest Seth

* Migrated keeper benchmark, kind of

* migrate keeper smoke tests

* bunch of optimisation

* migrate automation chaos test

* migrate log poller tests to Seth

* use anysynced key with log poller

* fix some of failing smoke tests

* fix go mod

* use correct Seth version

* better automation some test structure

* better automation some test structure

* go mod

* fix issues with missing LINK funds

* fix keeper test

* fix automation test

* clean up, comments, migrate keeper benchmark test

* mend

* use latest Seth

* cleanups, comments, etc

* clean ups, automation tests + keeper tests now always test multiple contracts

* use newer action version

* debug results when upkeeps fails

* run automation tests with post-failure debug

* run automation tests with post-failure debug

* better staleness report

* more debug for basic automation tests

* more assertions

* try with sleep

* increase performance check timeout to 3m

* separate config from consumer start

* more debug for performance chcek

* try with just 1 contract

* one more clean up

* run keeper tests in CI to test them

* solve automation load test registration failures

* subscribe to headers in benchmark test to observe upkeeps

* centralised ephemeral keys on testnet check, fixed keeper smoke tests, added root key buffer check

* do not fail, when on testnet and ephemeral keys are !=0, set them 0 instead

* don't fail in test config, display warning if eph addresses are set and set them to 0

* fix possible nil pointer, remove block range timeout, mutex for error appending

* update seth version

* use buffered channel to read subscription headers from EVM node

* benchmark test increase RR mem and set DEBUG log level

* use CTF's concurrent executor to deploy contracts in parallel

* change architecture so that we have only 1 subscription and then we fan out the header to all goroutines; use SafeEVMHeader

* benchmark test increase geth disk and gaslimit

* decrease log level, latest seth and ctf

* update CTF dependency

* use concurrent executor in automation load test

* remove local replace

* add header subscription reconnect handling

* print return funds error

* limit automation load test deployment concurrency

* latest Seth + fix gun

* do not run keeper smoke tests on each commit anymore

* add one more error message

* use tagged CTF

* use latest Seth

* use latest Seth

* fix go.mod

* forget about smart reconnect, just do it in a loop

* use latest Seth

* check if context has error before executing any of contract calling wrapper functions

* prettify code + latest Seth

* update Seth

* optional retrying of read-only contract calls; used only by keeper benchmark test

* update Seth, estimate gas for funds sending

* fix new automation test, update Seth to v1.0.7

* increase timeout

* fix max price smoke test

* fix CreateOCRKeeperJobs for v2.0

* remove resolved TODO

* code review changes

* code review changes

* code review changes vol 2

* use CTF release version

* bump mod version

---------

Co-authored-by: anirudhwarrier <12178754+anirudhwarrier@users.noreply.github.com>

* fix issue introduced by merge conflict

* adjust keeper & automation TOML configs

* restore old integration-tests workflow that doesn't run keeper tests

* fix double log stream shutdown

* add back clNodeOpts

* use tagged CTF version

---------

Co-authored-by: Ilja Pavlovs <ilja.pavlovs@gmail.com>
Co-authored-by: anirudhwarrier <12178754+anirudhwarrier@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants